FRPA: A Framework for Recursive Parallel Algorithms

Recursion continues to play an important role in high-performance computing. However, parallelizing recursive algorithms while achieving high performance is nontrivial and can result in complex, hardto-maintain code. In particular, assigning processors to subproblems is complicated by recent observations that communication costs often dominate computation costs. Previous work [1]–[3] demonstrates that carefully choosing which divide-and-conquer steps to execute in parallel (breadth-first steps) and which to execute sequentially (depth-first steps) can result in significant performance gains over naïve scheduling. Our Framework for Recursive Parallel Algorithms (FRPA) allows for the separation of an algorithm’s implementation from its parallelization. The programmer must simply define how to split a problem, solve the base case, and merge solved subproblems; FRPA handles parallelizing the code and tuning the recursive parallelization strategy, enabling algorithms to achieve high performance. To demonstrate FRPA’s performance capabilities, we present a detailed analysis of two algorithms: Strassen-Winograd [1] and CommunicationOptimal Parallel Recursive Rectangular Matrix Multiplication (CARMA) [3]. Our single-precision CARMA implementation is fewer than 80 lines of code and achieves a speedup of up to 11× over Intel’s Math Kernel Library (MKL) [4] matrix multiplication routine on “skinny” matrices. Our double-precision StrassenWinograd implementation, at just 150 lines of code, is up to 45% faster thanMKL for large square matrix multiplications. To show FRPA’s generality and simplicity, we implement six additional algorithms: mergesort, quicksort, TRSM, SYRK, Cholesky decomposition, and Delaunay triangulation [5]. FRPA is implemented in C++, runs in shared-memory environments, uses Intel’s Cilk Plus [6] for task-based parallelism, and leverages OpenTuner [7] to tune the parallelization strategy.